Skip to main content

Visual Camera Inference Node

Summary​

The Node is used to detect lanemarkings in the images produced from the Basler Dart Camera on the vehicle. This is currently implemented using Lanenet, an implementation of this paper. Lanenet uses a single encoder, dual decoder architecture. The encoder computes features vectors and both decoders use these features to compute a binary segmentation and an instance segmentation. These two segmentations are then combined using the DBScan clustering algorithm to produce the result. The resulting image is then transformed into a top view perspective and send to enviroment model.

Example GIF​

This GIF flashes switches between the Input Image and the resulting lane detections (before topview transformation). This demonstrations is close to real world and contains some errors, as in some frames both rightmost lanes are combined into one class. vci_demo.gif

Structure​

The internal operations of the node can be divided into roughly three stages. As can be seen in the following graphic:

GraphicsTVCINode.png

  1. In the first stage, the node receives its input image via ROS2 either from the Adapter for the Basler Dart Camera or from the Simulation Adapter. This image is then downsampled using CV2 to 512x256 pixels, which is the input size for Lanenet.
  2. In the second stage, Lanenet then processes the Image and generates a feature vector using its encoder stage. The Lanenet implementation we use allows for two different encode stages, bisnetv2 and vgg16. These features are then used by the two decoder stages to generate the instance segmentation image, which is used to differentiate between the different lanes in the image and binary segmentation which focusses on differentiating between the background and lanes.
  3. In the last stage, the two segmentation images are combined using the DBScan clustering algorithm and transformed using cv2 function into a Top-View Image. This is useful, as processing this image is easier for the enviroment model, to which the image is then sent.

References​

  1. Lanenet Repository
  2. BiseNetv2 Repository
  3. Lanenet Paper
  4. Tensorflow

Training the Network​

In order to train or retrain (transfer learning) Lanenet using our own training data, please refere to this repository: training-tensorflow-lanenet